Cross-Validation

Introduction

• Challenges in Cross-Validation: Traditional methods like k-fold and leave-one-out cross-validation face limitations like overfitting, variance, and inefficiency in large datasets.
• Innovative Solutions: New techniques such as Cross-Validation Voting (CVV), Monte Carlo cross-validation (MCCV), and improved methods for selecting the right number of components (Wold, EK procedures) enhance model generalization, reduce overfitting, and improve error rate estimation.
• Performance and Efficiency: Leveraging advanced methods like Parallel-CV and GPU-based parallelization, as well as alternative cross-validation approaches in multivariate models, significantly boosts efficiency and accuracy, especially for large or small datasets.

Literature Review

• CNN models face challenges like overfitting and generalization issues in supermarket product classification.[1]
• Ensemble learning strategies such as voting, boosting, and bagging are commonly used to improve performance but often rely on single classifiers or validation sets.
• The Cross-Validation Voting (CVV) method proposed by Duque Domingo et al. improves generalization and reduces overfitting, outperforming traditional methods in grocery classification.[2],[3],[4]

Literature Review

• Overview of Cross-Validation Methods: The paper covers data splitting techniques, including single hold-out, k-fold, and leave-one-out cross-validation. [5]
• Pros and Cons: Each method’s advantages and limitations are discussed in relation to dataset size and complexity.
• Key Concepts: The paper addresses overfitting, stratification, and how to select the final model based on cross-validation results. [[6]][7]

Literature Review

• Focus on Model Generalizability: The paper highlights the issue of overfitting in statistical models and their poor performance on new data. [8]
• Cross-Validation Techniques: It introduces k-fold and Monte Carlo cross-validation as methods to assess and improve generalizability, with practical implementation in R using the caret package.
• Interactive Example: A hands-on Shiny app example illustrates how factors like model complexity, sample size, and effect size impact generalizability.

Literature Review

• Cross-Validation in PCA: The paper examines how cross-validation techniques help select the correct number of components in principal component analysis (PCA) to avoid overfitting and improve prediction accuracy.[9]
• Alternative Methods: It introduces the improved Wold and Eastment–Krzanowski (EK) procedures to address overfitting and bias in component models.
• Performance of Methods: The study finds that the Eigenvector and EM methods outperform others, particularly for smaller datasets, emphasizing the importance of choosing the right cross-validation method based on dataset size and complexity.

Introduction to the Dataset

Martensite Starting Temperature

• Materials Science Dataset about Steel
• Martensite Starting Temperature (Ms in degree Celsius) & chemical elements (weight percent)
• Depending on the chemistry of a steel, Ms changes
• It is important as it controls strength of Steel
• The data has 16 variables for 1543 observations

Application

Dataset

• Data exploration

• Key variables (Table 1)

Methodology

• Modeling Approach:

  • Untransformed Model: Directly modeled Ms using predictors like C, Mn, Ni, Si, Cr, with interaction terms.

  • Log-Transformed Model: Modeled log(Ms) to handle non-normality and stabilize variance, using the same predictors and interaction terms.

  • Model Improvements (Predictors’ Removal, Introducing Interaction Parameters, Outliers’ Removal)

  • Model Diagnostics (ANOVA, AIC, Cross-Validation, Check for Multicollinearity, Influential Points’ Removal)

  • Model Evaluation: The log-transformed model showed significantly better performance with a lower AIC and cross-validation MSE. Residual deviance and cross-validation confirmed that the log model generalized better to unseen data.

• Cross-Validation Refinement:
  • K-Fold Cross-Validation with More Folds
  • Leave-One-Out Cross-Validation (LOOCV)
• Programing has been done by R [10] in Rstudio (version 2024.04.2)
• Utilized packages: tidyverse [11], classpackage [12], ggplot2 [13], psych [14], and boot [16]